Adaptive Dynamics of Realistic Small-World Networks

Continuing in the steps of Jon Kleinberg's and others celebrated work on decentralized search in small-world networks, we conduct an experimental analysis of a dynamic algorithm that produces small-world networks. We find that the algorithm adapts robustly to a wide variety of situations in realistic geographic networks with synthetic test data and with real world data, even when vertices are uneven and non-homogeneously distributed. We investigate the same algorithm in the case where some vertices are more popular destinations for searches than others, for example obeying power-laws. We find that the algorithm adapts and adjusts the networks according to the distributions, leading to improved performance. The ability of the dynamic process to adapt and create small worlds in such diverse settings suggests a possible mechanism by which such networks appear in nature

Concept-aware clustering for decentralized deep learning under temporal shift

Decentralized deep learning requires dealing with non-iid data across clients, which may also change over time due to temporal shifts. While non-iid data has been extensively studied in distributed settings, temporal shifts have received no attention. To the best of our knowledge, we are first with tackling the novel and challenging problem of decentralized learning with non-iid and dynamic data. We propose a novel algorithm that can automatically discover and adapt to the evolving concepts in the network, without any prior knowledge or estimation of the number of concepts. We evaluate our algorithm on standard benchmark datasets and demonstrate that it outperforms previous methods for decentralized learning.Comment: 4 pages, 2 figure

Multi-Document Summarization and Semantic Relatedness

Automatic summarization is the process of presenting the contents of written documents in a short, comprehensive fashion. Many approaches have been proposed for this problem, some of which extract content from the input documents (extractive methods), and others that generate the language in the summary based on some representation of the document contents (abstractive methods).This thesis is concerned with extractive summarization in the multi-document setting, and we define the problem as choosing the most informative sentences from the input documents, while minimizing the redundancy in the summary. This definition calls for a way of measuring the similarity between sentences that captures as much as possible of the meaning. We present novel ways of measuring the similarity between sentences, based on neural word embeddings and sentiment analysis. We also show that combining multiple sentence similarity scores, by multiplicative aggregation, helps in the process of creating better extractive summaries.We also discuss the use of information extraction for improving the quality of automatic summarization by providing ways of assessing the salience of information elements, as well as helping with the fluency of the output and providing the temporal dimension.Furthermore, we present graph-based algorithms for clustering words by co-occurrence, and for summarizing short online user-reviews by computing bicliques. The biclique algorithm provides a fast, simple algorithm for summarization in many e-commerce settings

Multi-Document Summarization and Semantic Relatedness

Automatic summarization is the process of presenting the contents of written documents in a short, comprehensive fashion. Many approaches have been proposed for this problem, some of which extract content from the input documents (extractive methods), and others that generate the language in the summary based on some representation of the document contents (abstractive methods).This thesis is concerned with extractive summarization in the multi-document setting, and we define the problem as choosing the most informative sentences from the input documents, while minimizing the redundancy in the summary. This definition calls for a way of measuring the similarity between sentences that captures as much as possible of the meaning. We present novel ways of measuring the similarity between sentences, based on neural word embeddings and sentiment analysis. We also show that combining multiple sentence similarity scores, by multiplicative aggregation, helps in the process of creating better extractive summaries.We also discuss the use of information extraction for improving the quality of automatic summarization by providing ways of assessing the salience of information elements, as well as helping with the fluency of the output and providing the temporal dimension.Furthermore, we present graph-based algorithms for clustering words by co-occurrence, and for summarizing short online user-reviews by computing bicliques. The biclique algorithm provides a fast, simple algorithm for summarization in many e-commerce settings

Representation learning for natural language

Artificial neural networks have obtained astonishing results in a diverse number of tasks.One of the reasons for the success is their ability to learn the whole task at once (endto-end learning), including the representations for data. This thesis will investigate representation learning for natural language through the study of a number of tasks ranging from automatic multi-document summarization to named entity recognition and the transformation of words into morphological forms specified by analogies.In the first two papers, we investigate whether automatic multi-document summarization can benefit from learned representations, and what are the best ways of incorporating learned representations in an extractive summarization system. We propose a novel summarization approach that represents sentences using word embeddings, and a strategy for aggregating multiple sentence similarity scores to compute summaries that take multiple aspects into account. The approach is evaluated quantitatively using the de facto evaluation system ROUGE, and obtains state-of-the-art results on standard benchmark datasets for generic multi-document summarization.The rest of the thesis studies models trained end-to-end for some specific tasks, and investigates how to train the models to perform well, and to learn internal representations of data that explain the factors of variation in the data.Specifically, we investigate whether character-based recurrent neural networks (RNNs) can learn the necessary representations for tasks such as named entity recognition (NER) and morphological analogies, and what is the best way of learning the representations needed to solve the mentioned tasks. We devise a novel character-based recurrent neural network model that recognize medical terms in health record data. The model is trained on openly available data, and evaluated using standard metrics on sensitive medical health record data in Swedish. We conclude that the model learns to solve the task and is able to generalize from the training data domain to the test domain.We then present a novel recurrent neural model that transforms a query word into the morphological form demonstrated by another word. The model is trained and evaluated using word analogies and takes as input the raw character-sequence of the words with no explicit features needed. We conclude that character-based RNNs can successfully learn good representations internally and that the proposed model performs well on the analogy task, beating the baseline with a large margin. As the model learns to transform words, it learns internal representations that disentangles morphological relations using only cues from the training objective, which is to perform well on the word transformation task.In other settings, such cues may not be available at training time, and we therefore present a regularizer that improves disentanglement in the learned representations by penalizing the correlation between activations in a layer. In the second part of the thesis we have proposed models and associated training strategies that solves the tasks and simultaneously learns informative internal representations; in Paper V this is enforced by an explicit regularization signal, suitable for when such a signal is missing from the training data (e.g. in the case of autoencoders)

Editing the simplest graphs

We study the complexity of editing a graph into a target graph with any fixed critical-clique graph. The problem came up in practice, in mining a huge word similarity graph for well structured word clusters. It also adds to the rich field of graph modification problems. We show in a generic way that several variants of this problem are in SUBEPT. As a special case, we give a tight time bound for edge deletion to obtain a single clique and isolated vertices, and we round up this study with NP-completeness results for a number of target graphs

Character-based recurrent neural networks for morphological relational reasoning

We present a model for predicting inflected word forms based on morphological analogies. Previous work includes rule-based algorithms that determine and copy affixes from one word to another, with limited support for varying inflectional patterns. In related tasks such as morphological reinflection, the algorithm is provided with an explicit enumeration of morphological features which may not be available in all cases. In contrast, our model is feature-free: instead of explicitly representing morphological features, the model is given a demo pair that implicitly specifies a morphological relation (such as write: writes specifying infinitive:present). Given this demo relation and a query word (e.g. watch), the model predicts the target word (e.g. watches). To address this task, we devise a character-based recurrent neural network architecture using three separate encoders and one decoder. Our experimental evaluation on five different languages shows that the exact form can be predicted with high accuracy, consistently beating the baseline methods. Particularly, for English the prediction accuracy is 94.85%. The solution is not limited to copying affixes from the demo relation, but generalizes to words with varying inflectional patterns, and can abstract away from the orthographic level to the level of morphological forms